<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <title>Prex Kernel Debugging Tips and Tricks</title>
  <meta content="text/html; charset=ISO-8859-1" http-equiv="content-type">
  <meta name="keywords" content="Prex, embedded, real-time, operating system, RTOS, open source, free">
  <meta name="author" content="Kohsuke Ohtani">
  <link rel="stylesheet" type="text/css" href="../default.css" media="screen">
  <link rel="stylesheet" type="text/css" href="../print.css" media="print">
</head>
<body>
<div id="top">
</div>
<div id="middle">

<table id="content" cellpadding="0" cellspacing="0">
  <tbody>

    <tr>
      <td id="header" colspan="2" valign="top">
        <table width="100%" border="0" cellspacing="0" cellpadding="0">
        <tr>
          <td id="logo">
            <a href="http://prex.sourceforge.net/">
            <img alt="Prex logo" src="../img/logo.gif" border="0"
            style="width: 250px; height: 54px;"></a>
          </td>
          <td id="brief" align="right" valign="bottom">
            An Open Source, Royalty-free,<br>
	    Real-time Operating System
          </td>
        </tr>
        </table>
      </td>
    </tr>

    <tr>
      <td id="directory" style="vertical-align: top;">
      <a href="http://prex.sourceforge.net/">Prex Home</a> >
      <a href="index.html">Document Index</a> >
      Kernel Debugging Tips and Tricks
    </tr>
    <tr><td class="pad" colspan="2" style="vertical-align: top;"></td></tr>

    <tr>
      <td id="doc" style="vertical-align: top;">
      <h1>Prex Kernel Debugging Tips and Tricks</h1>

<i>Version 1.1, 2008/09/25</i><br>

<h3>Table of Contents</h3>

<ul>
  <li><a href="#dbg">Building a Debugging Kernel</a></li>
  <li><a href="#print">Kernel Print</a></li>
  <li><a href="#panic">Kernel Panic</a></li>
  <li><a href="#assert">Assertion Check</a></li>
  <li><a href="#trace">Function Trace</a></li>
  <li><a href="#dump">Kernel Dump</a></li>
  <li><a href="#gdb">Remote debugging with GDB</a></li>
  <li><a href="#qemu">Using GDB with QEMU</a></li>
</ul>
<br>
<br>

<h2 id="dbg">Building a Debugging Kernel</h2>
<p>
The Prex kernel is compiled with debugging version by default.
The debugging version will have the following extra functions
compared with the release version.
<ul>
  <li>printf() to output the debug message.</li>
  <li>panic() log for the fatal error.</li>
  <li>ASSERT() for the assertion check.</li>
  <li>Kernel function trace.</li>
  <li>Dump of the kernel objects.</li>
</ul>
<p>
However, you should care about the following two points.
</p>
<ul>
  <li>The size of executable image will become larger than
      the release version.</li>
  <li>The execution speed becomes slower than the release version.
</ul>
<p>
In order to build a release version, you must set the shell variable
named NDEBUG to 1 before compiling the kernel.
</p>
<pre class="terminal">
$ export NDEBUG=1
</pre>


<h2 id="print">Kernel Print</h2>
<p>
The most powerful tool for kernel debugging is printf(). You can check the
code path, or get the data value by putting the printf() anywhere in 
the kernel code.
</p>
<p>
This is the subset of printf() in standard C library. You can use
only following identifiers for the output string.
</p>
<ul>
  <li>%d - Decimal signed integer</li>
  <li>%x - Hex integer</li>
  <li>%u - Unsigned integer</li>
  <li>%c - Character</li>
  <li>%s - String</li>
</ul>

<h2 id="panic">Kernel Panic</h2>
<h3>What is Kernel Panic?</h3>
<p>
Panic is used to detect unrecoverable error in the kernel.
The kernel code can call panic() routine anytime if it
encounters a fatal error. panic() function will stop the system and
display the message if output device is available. If the kernel is not
compiled with debug mode, panic() will just reset the system without
any message. When GDB is connect to the kernel, the control will be
dropped to the debugger in panic().
</p>

<h3>Panic Screen</h3>
The following is a sample screen of panic() on i386.
<pre class="terminal">
=====================================================
Kernel panic!
Failed to create interrupt thread
=====================================================
============================
Trap 3: Breakpoint
============================
Trap frame 80002f28 error 0
 eax 00000001 ebx 800110ac ecx ffffffff edx 800192b6 esi 00000000 edi 00000000
 eip 80014289 esp 80002f28 ebp 80002f78 eflags 00000002
 cs  00000010 ss  00000018 ds  00000018 es  00000018 esp0 00002fff
 >> interrupt is disabled
Stack trace:
 80011103
 8001130c
 80010115
</pre>

<h2 id="assert">Assertion Check</h2>

<h3>What is Assertion Check?</h3>
<p>
The assert macro puts error messages into the kernel.
If the specified expression in ASSERT() is false, it displays
the message to the output device and stops the system.
</p>
<p>
For example, the following ASSERT() macro is put to the entry of the
irq_attach() routine.
This means the caller of irq_attach() must provide a valid
function pointer for ISR (interrupt service routine).
</p>
<pre>
int irq_attach(int vector, int prio, int shared, int (*isr)(int), void (*ist)(int))
{
	ASSERT(isr != NULL);
	...
</pre>
If the kernel detect the invalid argument for isr, it will cause
a panic() with the following message.
<pre class="terminal">
Assertion failed: irq.c line:120 in irq_attach() 'isr != NULL'
</pre>
<p>
The important point is ASSERT() macro is only enabled with the kernel
debug version. If the assertion is caused by an application bug
or h/w bug, it should not be covered by ASSERT(). Such fault should be handled
as an "error" properly.
</p>

<h3>IRQ Assertion</h3>
<p>
There are some kernel routines that can not be called during
an interrupt context. For example, the following functions can not
be called by ISR.
</p>
<ul>
  <li>IRQ control code. (irq_attach() or irq_detach).</li>
  <li>The system call code that requires an user mode context.</li>
  <li>The kernel code that assumes it can synchronize by scheduling lock.
      (kmem).</li>
</ul>
<p>
In order to detect the invalid call of these functions, the kernel defines
a ASSERT_IRQ() macro. The kernel developer can put this macro in the 
head of the kernel routine which can not be called by ISR.
When ISR call such routine, the kernel will report the problem with panic()
message.
</p>

<h2 id="trace">Function Trace</h2>
The kernel trace is used to log the entry/exit of all functions at 
runtime.
It is useful to debug the timing critical issue related to 
an interrupt handler.

<h3>Installing the kernel trace</h3>
<p>
You must set the shell variable named KTRACE to 1 before compiling the kernel.
</p>
<pre class="terminal">
$ export KTRACE=1
</pre>
<p>
The kernel function trace is disabled by default. If you get the error
from gcc, you should check the function with "extern inline" in the header
file. Such function must be replaced as "static inline" routine to
enable the function trace.
</p>

<h3>Using the kernel trace</h3>
<p>
The following functions are available for the kernel trace.
</p>
<ul>
  <li>trace_on: Enable a function trace.</li>
  <li>trace_off: Disable a function trace.</li>
  <li>trace_dump: Dump the latest function log.</li>
</ul>
<p>
The trace code is using a ring buffer to store the log data.
So, older information will be disposed when newer log is filled in the buffer.
</p>

<h2 id="dump">Kernel Dump</h2>

<h3>Installing Kernel Dump</h3>
<p>
The kernel dump is used to dump the current status of each kernel
objects.
To install the kernel dump function, you must uncomment the following line
in config.h.
</p>
<pre>#define CONFIG_KDUMP 1</pre>
<p>
If you assign magic keys for the kernel dump, you can get the 
kernel dump as far as the keyboard IRQ is enabled.
The key assignment of the kernel dump depends on each platform.
</p>

<h3>Dump Sample</h3>
<p>
The following dump screen was got with Prex-i386pc.
</p>
<pre class="terminal">
Kernel dump usage:
F1=help F2=thread F3=task F4=object F5=timer F6=irq F7=dev F8=mem

thread_dump:
 mod thread   task     stat pol  prio base lock qntm total    susp sleep event
 --- -------- -------- ---- ---- ---- ---- ---- ---- -------- ---- ------------
 Knl 8002a014 800191a0 SLP  FIFO  022  022    1   50        1    0 interrupt
 Knl 80029a54 800191a0 SLP  FIFO  020  020    1   50        0    0 interrupt
 Knl 80029464 800191a0 SLP  RR    015  015    1   50        0    0 timer
 Knl 800190c0 800191a0 RUN* FIFO  255  255    1    0    44719    0 -
 Usr 8002a6e4 8002a574 SLP  RR    200  200    1   50        3    0 kbd

task_dump:
 mod task      nr_obj nr_thr map      susp exc hdlr name
 --- --------- ------ ------ -------- ---- -------- ------------
 Knl 800191a0*      1      4 8001ac20    0 00000000 kernel
 Usr 8002a574       1      1 8002a5e4    0 00000000 kmon

IRQ dump:
 vector isr      ist      ist-thr  ist-prio count
 ------ -------- -------- -------- -------- --------
 00     800110c8 00000000 00000000        0    54124
 01     80020784 800209f8 80029a54       20       66
 06     800212d0 80021360 8002a014       22        1
 12     80021aec 00000000 00000000        0        0

device_dump:
 device   open     close    read     write    ioctl    event    name
 -------- -------- -------- -------- -------- -------- -------- ------------
 8002a544 00000000 00000000 00000000 80020ecc 00000000 00000000 console
 80029fb4 80021b80 80021bb0 80021be0 00000000 00000000 00000000 mouse
 80029f44 8002161c 8002163c 80021714 80021800 800218dc 00000000 fd0
 800299e4 80020a30 80020a50 80020a70 00000000 00000000 00000000 kbd
 800299b4 00000000 00000000 80020584 00000000 00000000 00000000 rtc
 80029984 80022808 80022828 00000000 00000000 80022848 00000000 pm
 80029954 00000000 00000000 00000000 00000000 8002212c 00000000 cpu

page_dump:
 free pages:
 start      end      size
 --------   -------- --------
 00025000 - 00027000     2000
 00034000 - 0009f000    6b000
 00100000 - 040ff000  3fff000
 used=200K free=65971K total=66171K

kmem_dump: page=2 (8Kbyte) alloc=7056byte unused=1136byte
Allocated block:
 Size   48 bytes => 11
 Size   64 bytes => 5
 Size  112 bytes => 1
 Size  224 bytes => 4
 Size 1040 bytes => 5
Free block:
 Size   32 bytes => 1
 Size 1072 bytes => 1

vm_dump:
task=8002a574 map=8002a5e4 name=kmon
 region   virtual  physical size     flags
 -------- -------- -------- -------- -----
 8002a624 08048000 0002c000     1000 R----
 8002a654 08049000 0002e000     1000 RW---
 8002a6b4 7ffff000 0002f000     1000 RW---
 *total=12K bytes
</pre>

<h2 id="gdb">Remote Debugging with GDB</h2>

<h3>Installing the GDB remote stub</h3>
<p>
To install GDB stub into the Prex kernel, you must uncomment the following line
in config.h.
</p>
<pre>#define CONFIG_GDB 1</pre>
<p>
The GDB support is disabled by default.
</p>

<h3>Connecting to the remote machine</h3>
You should specify the path to the symbol file.

<pre class="terminal">(gdb) symbol-file /usr/src/prex/sys/prex.sym
(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0</pre>

<h3>Debugging Sample</h3>
<pre class="terminal">$ i386-elf-gdb
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "--host=i686-pc-cygwin --target=i386-elf".
(gdb) symbol-file /usr/src/prex-0.1.2/sys/prex.sym
Reading symbols from /usr/src/prex-0.1.2/sys/prex.sym...done.
(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0
Remote debugging using /dev/ttyS0
gdb_init () at gdb_glue.c:98
98      }
(gdb) b context_init
Breakpoint 1 at 0x80010d27: file context.c, line 72.
(gdb) info break
Num Type           Disp Enb Address    What
1   breakpoint     keep y   0x80010d27 in context_init at context.c:72
(gdb) c
Continuing.

Breakpoint 1, context_init (ctx=0x8001a16c, kstack=0x8002c414) at context.c:72
72      {
(gdb) bt 3
#0  context_init (ctx=0x8001a16c, kstack=0x8002c414) at context.c:72
#1  0x800137fe in thread_init () at thread.c:469
#2  0x800125e0 in kernel_main () at main.c:76
(More stack frames follow...)
(gdb) list context_init
67       *
68       * @ctx: context id (pointer)
69       * @kstack: kernel stack for the context
70       */
71      void context_init(context_t ctx, void *kstack)
72      {
73              struct kern_regs *k;
74              struct cpu_regs *u;
75
76              ctx->uregs = (struct cpu_regs *)(kstack - sizeof(struct cpu_regs
));
(gdb) up
#1  0x800137fe in thread_init () at thread.c:469
469             context_init(&amp;idle_thread.context, stk + KSTACK_SIZE);
(gdb) list
464                     panic("Failed to allocate idle stack");
465             memset(stk, 0, KSTACK_SIZE);
466             idle_thread.kstack = stk;
467             printf("kernel stack size=%d\n", KSTACK_SIZE);
468
469             context_init(&amp;idle_thread.context, stk + KSTACK_SIZE);
470             list_insert(&amp;kern_task.threads, &amp;idle_thread.task_link);
471     }
(gdb) b 465
Breakpoint 2 at 0x800137c6: file thread.c, line 465.
(gdb) c
Continuing.
</pre>



<h2 id="qemu">Using GDB with QEMU</h2>

QEMU has capability to connect to GDB on the same local machine.

<ul>
<li>
<b>Preparing gdbinit</b><br>

You have to prepare the following "gdbqemu" for gdb.
<pre>
symbol-file /usr/src/prex/sys/prex.sym
target remote localhost:1234
</pre>
</li>

<li>
<b>Run QEMU</b><br>

Execute the following command:

<pre class="terminal">
$ qemu -s -S -fda (your directory)/prex-X.X.X.i386-pc.img -localtime
</pre>

Then, the following message will be appered.

<pre class="terminal">
Waiting gdb connection on port 1234
_
</pre>

</li>

<li>
<b>Run GDB</b><br>

<pre class="terminal">
$ gdb -x gdbqemu
</pre>


</li>

      </td>
    </tr>
    <tr>
      <td id="footer" colspan="2" style="vertical-align: top;">
        <a href="http://sourceforge.net">
        <img src="http://sourceforge.net/sflogo.php?group_id=132028&amp;type=1"
        alt="SourceForge.net Logo" border="0" height="31" width="88"></a><br>
        Copyright&copy; 2005-2008 Kohsuke Ohtani
      </td>
    </tr>

  </tbody>
</table>

</div>
<div id="bottom"></div>

</body>
</html>
